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Abstract 

The present paper examined published research in training and 
development to determine effect size reporting practices. 
Specifically, we evaluated the reporting of corrected effect 
sizes in research articles using multiple regression analyses. 
When possible we calculated corrected effect sizes and 
determined if the associated shrinkage could have impacted 
researcher interpretations. 
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R 2 Shrinkage in Multiple Regression Research: An Empirical 
Evaluation of Use and Impact of Adjusted Effect Formulae 

The reporting and interpretation of effect sizes is 
critical to good statistical practice. As the American 
Psychological Association (APA) Task Force on Statistical 
Inference noted (Wilkinson & APA Task Force on Statistical 
Inference, 1999), 

It is hard to imagine a situation in which a dichotomous 
accept-re j ect decision is better than reporting an actual 
p-value or, better still, a confidence interval. . . . 

Always provide some effect size estimate when reporting a 
p-value. (p. 599, emphasis added) 

The Task Force went on to state, "Always present effect sizes 
for primary outcomes. ... It helps to add brief comments that 
place these effect sizes in a practical and theoretical context" 
(p. 599, emphasis added) . 

The mandate to "always" report effect sizes is a 
substantial step beyond the fourth edition of the APA's 
Publication Manual, which only recommended reporting of effect 
sizes in research (APA, 1994, p. 18) . Empirical studies, 
however, have shown that this recommendation had little impact 
on researchers' inclusion of effect size information in their 
articles and even less impact on consultation of effects for 
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practical and theoretical context (cf. Henson & Smith, 2000; 
Vacha-Haase, Nilsson, Reetz, Lance, & Thompson, 2000). 

Influenced by the Task Force report, the recent fifth 
edition of the APA Manual (APA, 2001) called the "failure to 
report effect sizes" a "defect in the design and reporting of 
research" (p. 5) . The Manual later observed: "For the reader to 
fully understand the importance of your findings, it is almost 
always necessary to include some index of effect size or 
strength of relationship in your Results section" (p. 25). 

Of course, there are many different types of effect size 
indices. However, most can be grouped as (a) standardized mean 
differences (e.g., Cohen's d) or (b) variance-accounted-f or 
effects (e.g., eta 2 , r 2 , R 2 ) . Many researchers are familiar with 
the squared multiple correlation, or R 2 , often reported with 
multiple regression results. 

Furthermore, researchers can choose from uncorrected or 
corrected variance-accounted-f or effects. Because general linear 
model analyses, such as multiple regression, maximize shared 
variance between variables or sets of variables, these analyses 
capitalize on the sampling error variance in any given sample to 
yield the largest possible effect size. This sampling error 
variance is unique to the sample and unlikely to be present in 
future samples or the population. Accordingly, these indices 
(e.g., R 2 ) are called uncorrected effects sizes. 
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Corrected effects, however, "shrink" the uncorrected 
indices based on the degree of sampling error present in a 
sample to better estimate the real effect in the population. 
Sampling error (a) increases as sample size decreases, (b) 
increases as the number of variables in the model increases, and 
(c) increases as the theoretical population effect decreases. 
There are numerous corrected effect indices such as omega , 
Adjusted R 2 , epsilon 2 , and others. Snyder and Lawson (1992) and 
Yin and Fan (2001) review several corrected effect formulae. 

Each index somehow deals with the elements that increase 
sampling error, resulting in greater "shrinkage" and lower 
corrected estimates of the true population effect when greater 
amounts of sampling error are present. 

Because corrected effects are better models of the true 
population effect, they likely give better information as 
regards the replicability of one's results (a hallmark of good 
scientific practice) . In a Monte Carlo study, Yin and Fan (2001) 
demonstrated the accuracy of several corrected effect formulae. 

Given the general lack of effect size reporting in the 
literature, one might expect even less frequent reporting of 
corrected effects. Therefore, the purpose of the present paper 
was to empirically evaluate the reporting of corrected effects 
in three human resource, training, and development journals. 
Because effects are more commonly reported with multiple 
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regression analyses, we examined only regression applications. 

The paper will (a) document the frequency of corrected 
effect reporting in the journals examined, (b) identify the 
types of corrected effects reported, (c) and estimate the degree 
of shrinkage present, across studies for various corrected 
effect formulae presented by Snyder and Lawson (1992) and Yin 
and Fan (2001) when authors did not report corrected effects. To 
accomplish the last objective, we calculated several corrected 
effects using the information provided by authors. This allowed 
comparison of the possible differences in result interpretation 
depending on what type of effect was used. It also allowed for 
empirical investigation of the degree of correction resulting 
from the corrected effect formulae. 

Method 

We examined three human resource development journals. 

Human Resource Development International , Human Resource 
Development Quarterly and International Journal Of Training and 
Development . These publications were examined from 1998 to 2001, 
which totals 16 issues each. Table 1 shows the breakdown of 
reported effects. 



INSERT TABLE 1 ABOUT HERE 

Articles were evaluated to determine if they provided 
sufficient information to calculate corrected effect sizes when 
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they were not provided. In 51 of the 118 instances where 
uncorrected effects were reported enough information was 
provided to calculate a corrected effect size. Corrected effect 
sizes were calculated for these cases using two different 
formulae, drawn from Yin and Fan (2001): (a) the Pratt formula 

(see Claudy, 1978) and (b) the Ezekiel (1930) formula. The 
Ezekiel formula is referred to as Wherry-1 in Yin and Fan 
(2001). In Yin and Fan's (2001) simulation study, the Pratt (see 
Claudy, 1978) formula yielded the most accurate R 2 adjustments 
under several study conditions. The Ezekiel formula (1930) is 
often employed by major statistical software packages, including 
SPSS. 

Although there are some differences in efficiency of the 
corrected effect sizes using the formulae offered by Yin and Fan 
(2001), they point out, "the Pratt formula stands out as the 
best performer among the 6 formulas, because it performed best 
under three multicollinearity conditions (unbiased estimates 
about 91-98% of the time) , under three population p 2 conditions 
(unbiased estimates about 93-96% of the time) , and under five 
N/p ratio conditions (unbiased estimates 83-100% of the time)" 

(p. 214). 

Results and Discussion 

The reported effect sizes along with the calculated 
adjusted effect sizes can be seen in Table 2. Table 2 also gives 
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the N and number of predictors in each case, both of which 
impact the degree of R 2 shrinkage. Further, the amount of 
shrinkage for each formula is given along with the average of 
both estimates. 



INSERT TABLE 2 ABOUT HERE 
Reporting and Interpretation of Effects 

Human Resource Development International yielded two 
articles that used multiple regression analysis. 

One article reported both effect sizes and corrected effect 
sizes. The other article reported only the uncorrected effect 
size . 

Human Resource Development Quarterly articles utilized 
regression analysis on a more frequent basis. There were 
nineteen articles found to have used regression analyses during 
the time period examined. Out of the nineteen, eight reported 
corrected effect sizes, two of those eight reported only the 
corrected effects. One of the nineteen articles reported only 
the change in r 2 values for the stepwise regression, with no 
reference to overall effect sizes. 

International Journal of Training and Development provided 
six articles using regression analysis during the period 
evaluated. Only one of the articles reported corrected effect 
sizes. One other article reported only the regression weights 
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without reporting either corrected or uncorrected effect sizes. 

Overall, out of the 27 articles reviewed, 10 articles 
reported corrected effect sizes (37%). In most cases effect size 
was listed as a part of the statistical analysis, generally in a 
summary table. There were a few instances where the authors 
described the effect size as part of their narrative, such as 
reported by Hartel, Douthitt, Hartel, and Douthitt (1999): 
"Seventeen percent of the variance in the mean criterion score 
was accounted for by the IEI score" (pp. 81-82) . There were no 
instances where any author interpreted the effect size with 
regard to practical or theoretical relevance in their study. 

R 2 Shrinkage 

As reported in Table 2, shrinkage was calculated for all 51 
instances where sufficient information was provided by the 
authors. On average across the two formulae, shrinkage ranged 
from 0.5% to 30.8%. At the high end. Church and Waclawski's 
(1999) R 2 shrinkage was calculated to be between 26.6% and 30.8%, 
due primarily to the fact that they used 53 predictors. The use 
of so many predictors increased the likelihood of increased 
sampling error. 

In their article, Hanpachern, Morgan, and Griego (1998) 
stated, "Although these predictors are statistically 
significant, they do not account for a high proportion of the 
variance" (p. 344) . This conclusion would be further supported 
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had the adjusted effect (13.8%) been interpreted rather than the 
uncorrected effect (17%) . 

Ensher, Grant-Vallone, and Donaldson (2001) conducted two 
stepwise regressions. In the first analyses they reported, 
"Overall, the model explained a significant amount of the 
variance in job satisfaction (R 2 =.25, F=10.84, p<.001)" (p. 64). 

The corrected Pratt effect was 23.6%. This shrinkage may not 
have resulted in Ensher et al. changing their conclusions. 
However, in their second regression analysis they report similar 
findings with an uncorrected effect size of 21%. The calculated 
Pratt effect size for this analysis was 16.3%. This effect could 
have resulted in a different interpretation. 

Tansky and Cohen (2001) conducted a series of regression 
analyses, and in two instances they used stepwise regression. 

The first analysis resulted R 2 =0.41, with an adjusted effect of 
0.39 (Pratt). Given the magnitude of this effect there seems to 
be little impact, in this case, on interpretation for the two 
effects. The second regression analysis resulted in an 
uncorrected effect size of R 2 =0.21. The adjusted effect was 0.19 
(Pratt) . This shrinkage likely would make little difference in 
the interpretation of the data. 

Overall, the likely impact of shrinkage on interpretation 
varied, ranging from minimal difference between estimates to 
dramatic shrinkage resulting in negative R 2 values. Of course. 
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the ability to externally evaluate impact on interpretation is 
limited by a lack of context for the study. The authors 
themselves are best placed to evaluate this impact, had 
corrected effects been examined. 

Sampling Error and Generalizability 

As noted, sampling error represents the degree a given 
sample is not representative of the population. For any sample 
short of measuring an entire population, some sampling error is 
present that potentially limits the generalizability of one's 
results. Sampling error in multiple regression analyses is 
affected by several factors, including sample size, the number 
of predictors, and the theoretical population effect. 

Sample size is one of the major factors influencing 
sampling error. The greater the sample size, the smaller the 
sampling error. Because a larger sample size is closer to the 
actual population size, it more closely estimates the 
population. If the sample consisted of the entire population 
there would be zero error (of course, we would no longer call it 
a sample as well) . Conversely, the smaller the sample size the 
greater the error. 

The number of predictors also affects sampling error. As 
the number of predictors increases, the degree of sampling error 
theoretically increases. This is due to the fact that 
introducing more predictors introduces more opportunities for 
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error to be accumulated. For example, a given person may have 
well-measured scores on two predictors, but when a third 
variable is added, the person's score is not well-measured and 
is partially due to error. This source of sampling error speaks 
to the need to have a well-specified model, and to only examine 
those variables for which one has a theoretical rationale to 
examine . 

Finally, the larger the theoretical population effect, the 
less sampling error we expect in the sample. This source of 
error is less intuitive to understand. However, consider the 
case when the true population effect is R 2 =1.00. In this case, it 
does not matter who you sample from the population, all sample 
effects will also be 1.00. Conversely, if the population effect 
was R 2 =0. 00, you could get sample .effects ranging anywhere from 
0.00 to 1.00, depending on who was sampled and the relationship 
of their predictor scores to the dependent variable. In the 
later case, we expect more sampling error because the results 
are less trustworthy. 

Of course, we never know what the true population effect 
is. Therefore, the corrected effect formulae tend to use the 
observed sample effect as an estimate of the population effect. 

A more reasonable approach might be to include the meta-analytic 
average of the effects obtained in other studies (along with 
your study) as the best estimate of the true population effect. 
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Conclusion 

The APA Task Force (Wilkinson & APA Task Force on 
Statistical Inference, 1999) emphasized, "We must stress again 
that reporting and interpreting effect sizes in the context of 
previously reported effects is essential to good research. It 
enables readers to evaluate the stability of results across 
samples, designs, and analyses" (p. 599). As suggested by 
Thompson and Snyder (1997), 

. . . explicitly and reflectively linking research results 

in a given study to the effect sizes in previous studies is 
also a vehicle for evaluating result replicability. This 
can be done prospectively by formulating null hypotheses 
incorporating specific parameter expectations derived from 
previous research, as against the contemporary practice of 
always testing hypotheses of no difference or of no 
relationship (i.e., what Cohen, 1994, described as "nil" 
hypothesis testing), (p. 80) 

Additionally, the Task Force (Wilkinson & APA Task Force on 
Statistical Inference, 1999) encouraged authors to "Always 
present effect sizes for primary outcomes" (p. 599). 

Although regression effects tended to be reported in the 
present review, the inclusion of corrected effects was less 
frequent. Further, these effect sizes, when reported, were 
rarely placed in some theoretical or practical context in 
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relation to previously observed effects. Generally, the reader 
is left to draw his/her own conclusion about the practical 
impact of the effect in the research. 
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Table 1 

Reported Corrected and Uncorrected Effects for Multiple 



Regression 


Analyses 






Journal 


# of Articles 


# of uncorrected 
effects reported 


# of corrected 
effects reported 


Totals 


27 


118 


45 


HRDI 


2 


17 


5 


HRDQ 


19 


82 


35 


I JTD 


6 


19 


5 




o 



2 
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Table 2 

Reported and Adjusted Effect Sizes 



Shrinkage 



R 2 


Pratt 


Ezekiel 


N 


No . of 
Predictors 


Pratt 


Ezekiel 


Average 


0 . 110 


0.105 


0.105 


552 


3 


0.005 


0.005 


0 . 005 


0.060 


0.055 


0.055 


552 


3 


0.005 


0.005 


0 . 005 


0.050 


0.045 


0.045 


552 


3 


0.005 


0.005 


0 .005 


0.070 


0.065 


0.065 


552 


3 


0.005 


0.005 


0 .005 


0.100 


0.095 


0.095 


552 


3 


0.005 


0.005 


0.005 


0.020 


0.015 


0.015 


549 


3 


0.005 


0.005 


0.005 


0.007 


0.002 


0.002 


549 


3 


0.005 


0.005 


0.005 


0.010 


0.005 


0.005 


549 


3 


0.005 


0.005 


0.005 


0.020 


0.015 


0.015 


549 


3 


0.005 


0.005 


0.005 


0.030 


0.025 


0.025 


552 


3 


0.005 


0.005 


0.005 


0.010 


0.005 


0.005 


552 


3 


0.005 


0.005 


0.005 


0.050 


0.045 


0.045 


552 


3 


0.005 


0.005 


0.005 


0 . 310 


0.273 


0.261 


30 


2 


0.037 


0.049 


0.043 


0.570 


0.556 


0.539 


30 


2 


0 . 014 


0.031 


0.023 


0. 170 


0.139 


0.137 


131 


5 


0.031 


0.033 


0.032 


0.320 


0.254 


0.251 


131 


12 


0.066 


0.069 


0.067 


0.128 


-0.161 


-0 . 157 


207 


51 


0.289 


0.285 


0.287 


0.210 


-0.064 


-0.062 


207 


53 


0.274 


0.272 


0.273 


0. 140 


-0.160 


-0.156 


207 


53 


0.300 


0.296 


0.298 


0.230 


-0.037 


-0.035 


207 


53 


0.267 


0.265 


0.266 


0.210 


-0.064 


-0.062 


207 


53 


0.274 


0.272 


0.273 


0 . 110 


-0.201 


-0 . 196 


207 


53 


0 . 311 


0.306 


0.308 


0 . 140 


-0.160 


-0.156 


207 


53 


0.300 


0.296 


0.298 


0.160 


-0.132 


-0.129 


207 


53 


0.292 


0.289 


0.291 


0.220 


-0.051 


-0.048 


207 


53 


0.271 


0.268 


0 .270 


0. 170 


0. 138 


0.135 


100 


4 


0.032 


0.035 


0.034 


0.070 


0.068 


0.068 


1250 


3 


0.002 


0.002 


0.002 


0 . 140 


0.120 


0 . 119 


123 


3 


0.020 


0.022 


0.021 


0.210 


0.163 


0.160 


67 


4 


0.047 


0.050 


0.048 


0.360 


0.354 


0.351 


146 


2 


0.006 


0.009 


0.007 


0.400 


0.395 


0.392 


146 


2 


0.005 


0.008 


0.007 


0 . 270 


0.2 62 


0.260 


146 


2 


0.008 


0.010 


0.009 
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